feat: runtime CP<->DP mode switching for xLLM workers. by cchh05 · Pull Request #1968 · xLLM-AI/xllm

cchh05 · 2026-07-17T08:09:11Z

Description

Add support for runtime CP<->DP mode switching on a live xLLM worker instance in ~500 ms without a service restart.

Startup builds BOTH ATB graphs (dual-mode) when --enable_runtime_cp_dp_switch=true, so a flip is a graph handle swap rather than a rebuild.
Flip flow: scheduler drains in-flight requests → pause workers → swap the active ParallelArgs (cp/dp) and block-manager pool atomically → resume.
Worker-side call sites read the live parallel_args_.cp_size() (not the immutable options_.cp_size()), so already-scheduled requests observe the new layout after resume.
New RPC surface: mode_switch.proto + ModeSwitchService, exposed via the xLLM server. Design doc lands under docs/en/features/runtime_cp_dp_switch.md.
Fully gated by --enable_runtime_cp_dp_switch=false (default). When off, worker startup and step paths are byte-identical to today.

Follow-on PR (#TBD) adds the AutoFlipController, the heal path, and bvar metrics.

Motivation

Traffic patterns rarely stay in one regime forever — long-context ingest bursts favor CP; concurrent short-request traffic favors DP. Static provisioning per instance forces the operator to overprovision one lane or accept degraded metrics on the other. Runtime CP<->DP mode switching lets an xLLM instance flip layout on demand, so a single deployment can adapt across the day rather than being statically sized for either lane.

Design highlights

Dual-graph startup: dual_parallel_args_ is attached to each worker at construction; layer implementations initialize 4 ATB nodes (cp_prefill, cp_decode, dp_prefill, dp_decode) that share atb_weight_tensors_ so there is no weight duplication. Resident cost is ~600 MB / NPU for the extra graph workspace.
Atomic pool swap: BlockManagerPool is guarded by a shared-ptr; the hot read path is lockless, and a flip swaps the handle. HierarchyBlockManagerPool rebuild only happens at pause time.
Runtime awareness at call sites: added uses_worker_cp_partition(const ParallelArgs&) overload that reads the live parallel_args_.cp_size(). options_.cp_size() stays frozen at startup; the overload is what callers on the hot path use post-flip.

Related Issues

Part of #1967.

Change Type

Pull Request Checklist

PR Title and Commit Messages

The PR title and each commit message follow the xLLM commit format: <type>: <subject>.

Pre-commit Checks

clang-format was run against all touched C++ files (verified via --dry-run --Werror, 0 violations).
Full pre-commit run --all-files — not run locally due to environment gap on my dev machine; happy to run whatever the CI does and iterate on any reported issue.

Self Review

I have self-reviewed the code according to .agents/skills/code-review/references/custom-code-style.md.
This PR is rebased onto upstream main at commit 52eca61a (immediate pre-PR#1938 base). See "Reviewer Notes" below for why not main HEAD.

Build and Test Coverage

Tests have been added: dual_parallel_args_test, atomic_pool_handle_test, lopsided_defer_gate_test.
CUDA python setup.py build test: not run — feature is NPU-only in this PR (no CUDA layer wiring). Behavior on CUDA is unchanged (feature gated off).
NPU: E2E verify_switch regression on 8*A3 DeepSeek V32 (61 layers), 18/18 flips stable, ~500 ms/flip. lm_eval CP == DP within tolerance. 10-min soak @ 32 procs, HBM watermark stable.
MLU: not run — feature is NPU-only in this PR. Behavior on MLU is unchanged (feature gated off).

Reviewer Notes

Base commit: this PR is based on upstream commit 52eca61a (immediate pre-PR#1938 base), not main HEAD, because PR#1938 introduces a laser_attention dependency that requires a MindIE-SD toolkit not yet in my dev environment. If reviewers prefer, I can rebase onto latest main once that toolkit gap is resolved — the feature code itself has no dependency on laser_attention.
Largest single file: xllm/core/layers/npu/npu_deepseek_v32_decoder_layer_impl.cpp (~150 lines added); most of it is symmetric to the existing 2-node init.
New proto: mode_switch.proto is a new file. No changes to existing protos.
Design doc: docs/en/features/runtime_cp_dp_switch.md — happy to iterate here on wording.
Not yet covered (documented in the design doc's "not yet covered" section):
- Cross-machine disagg PD flip (single-node validated).
- Long-run (>1 h) stability.
- Non-DeepSeek-V32 layer variants — the current PR wires only the V32 layer. Extending to other layer families is a follow-up.

…hing. Follow-on to the runtime CP<->DP mode switching feature. Adds: 1. AutoFlipController — a scheduler-facing controller that watches traffic stats and issues SwitchMode RPCs, so an instance can flip layout autonomously instead of relying on an external caller. 2. Heal path — after a switch, if a small tail of in-flight requests fails to converge (e.g. long-prefill sequences straddling the flip), a bounded heal window (default 4 rounds) drains them safely without holding the whole cluster. 3. CP -> DP pending gate — refuses a CP -> DP flip while the CP side still has queued long-prefill sequences that would be silently split apart by the DP layout. 4. 7 bvar metrics on `/vars`: xllm_mode_flip_total xllm_mode_flip_success_total xllm_mode_flip_target_cp_total xllm_mode_flip_target_dp_total xllm_mode_flip_latency_ms xllm_mode_flip_heal_rounds xllm_mode_flip_pending_gate_reject_total Testing ------- * Unit: `auto_flip_decide_target_test` (278 lines, covers stat window, minimum flip interval, target selection, hysteresis). * E2E: heal path exercised by the same `verify_switch` regression, which was extended to inject a long prefill straddling the flip; 842 ms average tail drain, 18/18 flips still convergent. * Soak: 10 min @ 32 procs with periodic auto-flips, HBM stable. Depends on xLLM-AI#1968 (feat: runtime CP<->DP mode switching).

XuZhang99 · 2026-07-17T10:25:37Z

please rebase the latest main branch, we have updated docs dir. Besides, plz add zh version docs.

…hing. Follow-on to the runtime CP<->DP mode switching feature. Adds: 1. AutoFlipController — a scheduler-facing controller that watches traffic stats and issues SwitchMode RPCs, so an instance can flip layout autonomously instead of relying on an external caller. 2. Heal path — after a switch, if a small tail of in-flight requests fails to converge (e.g. long-prefill sequences straddling the flip), a bounded heal window (default 4 rounds) drains them safely without holding the whole cluster. 3. CP -> DP pending gate — refuses a CP -> DP flip while the CP side still has queued long-prefill sequences that would be silently split apart by the DP layout. 4. 7 bvar metrics on `/vars`: xllm_mode_flip_total xllm_mode_flip_success_total xllm_mode_flip_target_cp_total xllm_mode_flip_target_dp_total xllm_mode_flip_latency_ms xllm_mode_flip_heal_rounds xllm_mode_flip_pending_gate_reject_total Testing ------- * Unit: `auto_flip_decide_target_test` (278 lines, covers stat window, minimum flip interval, target selection, hysteresis). * E2E: heal path exercised by the same `verify_switch` regression, which was extended to inject a long prefill straddling the flip; 842 ms average tail drain, 18/18 flips still convergent. * Soak: 10 min @ 32 procs with periodic auto-flips, HBM stable. Depends on xLLM-AI#1968 (feat: runtime CP<->DP mode switching).

cchh05 · 2026-07-17T10:37:16Z

Thanks for the review, @XuZhang99! Addressed in the latest push:

Moved to the new Astro docs structure: the doc now lives at docs/src/content/docs/en/features/runtime_cp_dp_switch.md with Starlight frontmatter (title + sidebar.order: 91, following the existing feature-doc convention).
Added the zh version: docs/src/content/docs/zh/features/runtime_cp_dp_switch.md, a full translation of the en version.
Cleaned up internal references: removed the pointer to my local MEMORY.md / debugging notes and generalized the verify_switch.sh reference (dropped the local absolute path).

Regarding the rebase-to-latest-main request: as noted in the PR description, this branch is based on upstream 52eca61a (immediate pre-#1938 base) because #1938 introduces the laser_attention dependency that requires a MindIE-SD toolkit not yet in my dev environment. I'd love to rebase onto main HEAD once I can build locally against that toolkit — happy to take a pointer if there's a recommended way to skip that dependency for CI-only validation. The docs move above works against either base, so this shouldn't block further review.

PR2 (#1969) has been rebased on top of the updated PR1 tip and force-pushed.

cchh05 · 2026-07-17T13:22:16Z

Update: rebased PR1 onto the current main HEAD (commit 5019610d at the time of rebase; will re-align if main moves further before merge). Force-pushed to pr1-cp-dp-switch-on-52eca61a, so this PR now includes:

Rebase conflict resolutions (4 files)

xllm/core/runtime/llm_worker_impl.cpp — kept the runtime-aware uses_worker_cp_partition(const ParallelArgs&) overload; three call sites now use parallel_args_ (live) instead of options_ (startup frozen) so a CP<->DP flip is respected on the logits and embeddings paths.
xllm/core/runtime/worker_impl.cpp — kept the unpack-then-resolve batch-type ordering, needed for the multi-node RPC path (packed input meta only becomes real after unpack; deciding CP prefill before unpack wrongly drove pure-decode into prepare_cp_prefill_inputs).
xllm/core/distributed_runtime/worker_server.cpp — merged upstream Platform CP-config log with the dual-mode-aware Worker constructor branch.
xllm/core/scheduler/continuous_scheduler.cpp — merged the upstream Platform::uses_model_cp_partition() gate with our active_dp_size_ > 1 runtime gate so live_cp_size reflects a flip.

Adaptations to upstream refactors (2 files, non-conflicting but still needed)

xllm/core/kernels/npu/npu_grouped_matmul.cpp — switched to upstream's current 22-arg npu_grouped_matmul signature (std::optional<int64_t> output dtype + 4 trailing c10::nullopts). The rebase auto-merged the wrong 18-arg version.
xllm/core/distributed_runtime/llm_engine.cpp — rebuild_block_manager_pool no longer instantiates HierarchyBlockManagerPool (upstream disabled it in refactor: delete prefix-cache upload feature and disable hierarchy host-mode (1/n). #1787). Falls back to BlockManagerPool in the normal path, and LOG(FATAL)s in the host_blocks_factor > 1.0 || enable_kvcache_store case exactly like upstream's init path does.

Docs move + zh version added earlier are still there. clang-format --dry-run --Werror clean on all touched files.

Local build validation attempted on 2 self-hosted A3/ARM boxes with the same cann9-20260605 image the CI uses; both are blocked by environment gaps (stale submodule pins for xllm_ops and xllm_atb_layers, cargo config layering) that don't apply to the upstream self-hosted runner. Would appreciate a CI run to confirm the rebase is clean end-to-end.

PR2 (#1969) will be re-based on top of the updated PR1 once this one settles.

…hing. Follow-on to the runtime CP<->DP mode switching feature. Adds: 1. AutoFlipController — a scheduler-facing controller that watches traffic stats and issues SwitchMode RPCs, so an instance can flip layout autonomously instead of relying on an external caller. 2. Heal path — after a switch, if a small tail of in-flight requests fails to converge (e.g. long-prefill sequences straddling the flip), a bounded heal window (default 4 rounds) drains them safely without holding the whole cluster. 3. CP -> DP pending gate — refuses a CP -> DP flip while the CP side still has queued long-prefill sequences that would be silently split apart by the DP layout. 4. 7 bvar metrics on `/vars`: xllm_mode_flip_total xllm_mode_flip_success_total xllm_mode_flip_target_cp_total xllm_mode_flip_target_dp_total xllm_mode_flip_latency_ms xllm_mode_flip_heal_rounds xllm_mode_flip_pending_gate_reject_total Testing ------- * Unit: `auto_flip_decide_target_test` (278 lines, covers stat window, minimum flip interval, target selection, hysteresis). * E2E: heal path exercised by the same `verify_switch` regression, which was extended to inject a long prefill straddling the flip; 842 ms average tail drain, 18/18 flips still convergent. * Soak: 10 min @ 32 procs with periodic auto-flips, HBM stable. Depends on xLLM-AI#1968 (feat: runtime CP<->DP mode switching).

Add support for flipping worker layout between CP_PREFILL (cp=N, dp=1) and DP_DECODE (cp=1, dp=N) on a live instance in ~500 ms, without a service restart. Traffic patterns rarely stay in one regime forever; this lets a single deployment adapt across the day rather than being statically sized for either lane. Design ------ * Startup builds BOTH ATB graphs (dual-mode) when `--enable_runtime_cp_dp_switch=true`, so a flip is a graph handle swap, not a rebuild. * Flip flow: scheduler drains in-flight requests -> pause workers -> swap the active `ParallelArgs` (cp/dp) and block-manager pool atomically -> resume. The block-manager pool is guarded by a shared-ptr for lockless reads on the hot path. * Worker-side call sites read the live `parallel_args_.cp_size()` (not the immutable `options_.cp_size()`), so already-scheduled requests observe the new layout after resume. New surfaces ------------ * `mode_switch.proto` + `ModeSwitchService`: RPC surface for flipping, exposed via the xLLM server. * `AutoFlipController` scaffolding for autoscaling callers. * `docs/en/features/runtime_cp_dp_switch.md`: user-facing design doc. Testing ------- * Unit: `dual_parallel_args_test`, `atomic_pool_handle_test`, `lopsided_defer_gate_test`. * E2E on 8*A3: `verify_switch` regression, 18/18 flips stable across 61 model layers; lm_eval CP == DP within tolerance; 10-min soak on 32-proc load did not regress HBM watermark. Related issue: xLLM-AI#1967

cchh05 · 2026-07-17T14:54:13Z

@XuZhang99 Thanks for the review!

Rebased onto the latest main HEAD (abe92b83, upstream had advanced 3 commits after my earlier rebase attempt). Only one content conflict — in xllm/core/distributed_runtime/disagg_pd_service_impl.cpp, where PR #1972 (DeepSeek-V4 PD disagg) refactored the block-collection path into a has_multi_block_export() branch and moved the shared_num/blocks declarations into the else arm. Resolved by keeping the upstream refactor and merging in the FLIPDIAG D_recv scheduled diagnostic log this PR needs. PR is now MERGEABLE.

Zh docs are in the branch — docs/src/content/docs/zh/features/runtime_cp_dp_switch.md matches the en version 1:1. Please take another look when you get a chance.

New head: 9140c533b05fe7e5a3e8b390e60020aa7a20c639.

cchh05 requested review from Clement-Wang26, DongheJin, DragonFive, JimHsiung, Kang-Meng, RobbieLeung, XuZhang99, liujinguang0125, liutongxuan, walsonyang, xiao-yu-chen, yingxudeng, yq33victor and zhang-minchao as code owners July 17, 2026 08:09

This was referenced Jul 17, 2026

Feature: Runtime CP<->DP mode switching for xLLM workers #1967

Open

feat: AutoFlipController + heal path + bvar metrics for CP<->DP switching. #1969

Open

cchh05 force-pushed the pr1-cp-dp-switch-on-52eca61a branch from 1b6a28c to 8ca601c Compare July 17, 2026 10:20

XuZhang99 reviewed Jul 17, 2026

View reviewed changes

cchh05 force-pushed the pr1-cp-dp-switch-on-52eca61a branch from 8ca601c to 99fa9f1 Compare July 17, 2026 10:37

cchh05 force-pushed the pr1-cp-dp-switch-on-52eca61a branch from 99fa9f1 to 4264e13 Compare July 17, 2026 13:21

cchh05 force-pushed the pr1-cp-dp-switch-on-52eca61a branch from 4264e13 to 9140c53 Compare July 17, 2026 14:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: runtime CP<->DP mode switching for xLLM workers.#1968

feat: runtime CP<->DP mode switching for xLLM workers.#1968
cchh05 wants to merge 1 commit into
xLLM-AI:mainfrom
cchh05:pr1-cp-dp-switch-on-52eca61a

cchh05 commented Jul 17, 2026

Uh oh!

XuZhang99 Jul 17, 2026

Uh oh!

cchh05 commented Jul 17, 2026

Uh oh!

cchh05 commented Jul 17, 2026

Uh oh!

cchh05 commented Jul 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

cchh05 commented Jul 17, 2026

Description

Motivation

Design highlights

Related Issues

Change Type

Pull Request Checklist

PR Title and Commit Messages

Pre-commit Checks

Self Review

Build and Test Coverage

Reviewer Notes

Uh oh!

XuZhang99 Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

cchh05 commented Jul 17, 2026

Uh oh!

cchh05 commented Jul 17, 2026

Uh oh!

cchh05 commented Jul 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants